System and method for ordering a print product including a digital image utilizing augmented reality

ABSTRACT

A method for providing a custom print product comprises displaying a video that includes an exhibited physical product; comparing feature or attribute data associated with the exhibited product with a feature or attribute associated with the available product; displaying a digital overlay in the video and in association with the displayed exhibited product if the displayed exhibited product corresponds to the available product; allowing for the selection of the digital overlay and then a digital image, wherein the selected digital image is not representative of the displayed exhibited physical product, and wherein the selected digital image is configured to be disposed on a selected physical product representative of the displayed exhibited physical product associated with the selected digital overlay to provide the customized print product.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.17/239,883, filed Apr. 26, 2021, now U.S. Pat. No. 11,356,568, which isa continuation of U.S. patent application Ser. No. 16/202,175, filedNov. 28, 2018, now U.S. Pat. No. 10,992,826, which in turn claims thebenefit of U.S. Patent Application No. 62/718,601, filed on Aug. 14,2018, the contents of which are incorporated by reference in theirentirety.

FIELD OF THE INVENTION

The present invention is directed to a system and method for providing acustom print product including a digital image utilizing augmentedreality; in particular, the present invention utilizes a method thatrecognizes an exhibited physical product being displayed in a video on adisplay of a mobile computing device, identifies a digital overlayassociated with the recognized exhibited physical product, displays theidentified digital overlay in association with the recognized exhibitedphysical product in the video on the display, and provides for theselection of the digital overlay to allow for the display of informationrelated to the exhibited physical product and ordering of a printproduct including a digital image selected using the mobile computingdevice.

BACKGROUND OF THE INVENTION

Photo kiosks are an interactive computer-based platform that may bephysically located within retail stores, drug stores and grocery stores(hereinafter “retail store location”). They are a convenient platformbecause they offer a customer the ability to upload digital imageslocated in the user's mobile computing device or access digital imagesthat were previously uploaded by the customer to the kiosk or thirdparty social media website, and customize a print product using theuploaded digital images without having to interact with store personnel.This allows the user to take the necessary time to select a printproduct without being pressured. Further, depending on the capabilitiesof the kiosk and the print product ordered by the consumer, the kioskmay include a printing device that allows for the fulfillment of theprint product on the spot. For example, a kiosk may be capable ofinstantly printing various sized image prints while the consumer waitsproviding instant satisfaction to the consumer.

While photo kiosks provide many advantages to the consumer, they doprovide some drawbacks. For instance, when a customer is using a kioskto select a product to associate with a digital image, the product istypically presented in digital form on a display of the kiosk. Viewingthe product in digital form could make it difficult for the customer toenvision the actual physical dimensions of the product and createhesitation on the part of the consumer to purchase the product. Thisresults in underutilization of kiosks. Also, there is a significant costin designing, manufacturing, and integrating a kiosk into aphotofinishing system. Kiosks include many of the same hardwarecomponents and peripherals that are in a typical computing device, withthe addition of an integrated printing device that is configured toproduce print products on the spot. It is not uncommon for kiosks to beplaced in multiple retail store locations for a given retail chain, andthe total cost of providing kiosks at each of these locations issubstantial. When a kiosk is installed in a retail store, it istypically placed in a high-traffic, premium location so that it canattract the attention of customers and be easily accessible for use.This premium floor space that the kiosk occupies is valuable andpotentially is taking away from the retailer's ability to sell otherproducts.

Accordingly there is a need for a system and method for ordering a printproduct while located in a retail store location without the need to usea kiosk. The present invention fills these needs as well as other needs.

SUMMARY OF THE INVENTION

In one aspect, the present invention includes a computer-implementedmethod programmed for execution in a computing environment for providinga custom print product. The method comprises: a) storing a digitalrepresentation of an available product in a memory; b) displaying avideo that includes an exhibited physical product (e.g., image print,calendar, mug, photo book, poster, blanket, phone case, pillow,clothing, or jewelry); c) comparing one or more feature or attributeassociated with the exhibited physical product being displayed in thevideo with feature data or attribute data associated with the availableproduct; d) upon determining that the displayed exhibited physicalproduct corresponds to the available product, displaying a digitaloverlay in the video and in association with the displayed exhibitedphysical product, wherein the digital overlay may include a descriptionof the displayed exhibited physical product; e) allowing for theselection of the digital overlay being displayed in the video; f) inresponse to the selection of the digital overlay, allowing for theselection of a digital image, wherein the selected digital image is notrepresentative of the displayed exhibited physical product, and whereinthe selected digital image is configured to be disposed on a selectedphysical product representative of the displayed exhibited physicalproduct associated with the selected digital overlay to provide thecustomized print product.

In another aspect, the present invention may utilize image and/or objectrecognition to compare the exhibited physical product being displayed inthe video with the digital representation of the available product. Inthe instance that image recognition is used, the feature data of theavailable product may include a stock digital image, wherein the one ormore feature of the exhibited physical product is a stock image, whereinstep c) includes scanning the displayed exhibited physical product toidentify the stock image disposed thereon. The displayed exhibitedphysical product corresponds to the digital representation of theavailable product when the stock image matches the stock digital image.

In the instance that object recognition is used, the attribute data mayinclude at least one dimension of the available product, wherein step c)includes scanning the displayed exhibited physical product to identifyat least one dimension of the displayed exhibited physical product. Thedisplayed exhibited physical product corresponds to the availableproduct when the at least one dimension of the displayed exhibitedphysical product matches the at least one dimension of the availableproduct. In addition, or alternatively to, using at least one dimension,the attribute data may include at least one shape of the availableproduct, wherein step c) includes scanning the displayed exhibitedphysical product to identify the at least one shape of the displayedexhibited physical product. The displayed exhibited physical productcorresponds to the available product when the at least one shape of thedisplayed exhibited physical product matches the at least one shape ofthe available product.

In another aspect, the step of displaying the digital overlay inassociation with the displayed exhibited physical product includessuperimposing the digital overlay on top of the displayed exhibitedphysical product, and/or above, below, right, left, or diagonal of thedisplayed exhibited physical product. Further, the digital overlay maybe displayed in the same perspective as the exhibited physical productis displayed in the video so that the digital overlay appears in thevideo in augmented reality. Therefore, as the camera is moved to changethe angle at which the exhibited product is being displayed on the userinterface of the mobile computing device, the perspective appearance ofthe digital overlay changes in unison with the perspective appearance ofthe associated exhibited physical product.

Further, the methods referred to above may be embodied in computerreadable instructions stored on a non-transitory computer readablemedium.

In yet another aspect, a computing system for providing a custom printproduct is provided. The system may comprise a first memory for storingcomputer-executable instructions, a camera configured for capturing avideo, a display configured for displaying the video, and a processorconfigured for executing the computer-executable instructions to performthe following steps: a) utilizing the camera to display a video thatincludes an exhibited physical product; b) comparing one or more featureor attribute associated with the exhibited physical product beingdisplayed in the video with feature data or attribute data associatedwith an available product, wherein the feature data or attribute data isstored in a server memory of a server, wherein the server is incommunication with the processor over a network; c) upon determiningthat the displayed exhibited physical product corresponds to theavailable product, displaying a digital overlay in the video and inassociation with the displayed exhibited physical product, wherein thedigital overlay may include a description of the displayed exhibitedphysical product; d) allowing for the selection of the digital overlaybeing displayed in the video on the display; e) in response to theselection of the digital overlay, allowing for the selection of adigital image to customize the displayed exhibited physical product,wherein the selected digital image is not representative of thedisplayed exhibited physical product, and wherein the selected digitalimage is configured to be disposed on a selected physical productrepresentative of the displayed exhibited physical product associatedwith the selected digital overlay to provide the customized printproduct.

In a further aspect, a computing system for providing a customized printproduct is provided. The system comprises a first memory for storingcomputer-executable instructions; a camera configured for capturing avideo; a display configured for displaying the video; and a processorconfigured for executing the computer-executable instructions to performthe following steps: a) utilizing the camera to display a video thatincludes an exhibited physical product; b) comparing one or more featureor attribute associated with the exhibited physical product beingdisplayed in the video with feature data or attribute data associatedwith an available product, wherein the feature data or attribute data isstored in the first memory of a server; c) upon determining that thedisplayed exhibited physical product corresponds to the availableproduct, displaying a digital overlay in the video and in associationwith the displayed exhibited physical product; d) allowing for theselection of the digital overlay being displayed in the video on thedisplay; and e) in response to the selection of the digital overlay,allowing for the selection of a digital image to customize the displayedexhibited physical product, wherein the selected digital image is notrepresentative of the displayed exhibited physical product, and whereinthe selected digital image is configured to be disposed on a selectedphysical product representative of the displayed exhibited physicalproduct associated with the selected digital overlay to provide thecustomized print product.

In another aspect, a computing system for providing a customized printproduct is disclosed herein. The system comprises a first memory forstoring computer-executable instructions; a camera configured forcapturing a video; a display configured for displaying the video; and aprocessor configured for executing the computer-executable instructionsto perform the following steps: a) utilizing the camera to display avideo that includes an exhibited physical product; b) comparing one ormore feature or attribute associated with the exhibited physical productbeing displayed in the video with one or more feature or attributeassociated with a digital representation of an available product storedin the first memory; c) upon determining that the displayed exhibitedphysical product corresponds to the digital representation of theavailable product, displaying a digital overlay in the video and inassociation with the displayed exhibited physical product; d) allowingfor the selection of the digital overlay being displayed in the video onthe display; and e) in response to the selection of the digital overlay,allowing for the selection of a digital image to customize the displayedexhibited physical product, wherein the selected digital image is notrepresentative of the displayed exhibited physical product, and whereinthe selected digital image is configured to be disposed on a selectedphysical product representative of the displayed exhibited physicalproduct associated with the selected digital overlay to provide thecustomized print product.

BRIEF DESCRIPTION OF THE DRAWINGS

The above-mentioned and other features and advantages of this invention,and the manner of attaining them, will become apparent and be betterunderstood by reference to the following description of the invention inconjunction with the accompanying drawing, wherein:

FIG. 1 is a system diagram generally illustrating various componentsthat may be utilized for the implementation of one or more aspects ofthe present invention;

FIG. 2 is a flow chart illustrating an exemplary method for implementingone or more aspects of the present invention;

FIGS. 3-12 are exemplary screen shots from a mobile computing devicewhen used in accordance with the method set forth in FIG. 2; and

FIG. 13 is a block diagram generally illustrating a computingenvironment in which the invention may be implemented.

DETAILED DESCRIPTION OF THE INVENTION

Generally, the systems and methods described herein for ordering a printproduct including a digital image using augmented reality may beimplemented in hardware, software or a combination thereof, and may bedistributed across a variety of computing devices.

This document is organized as follows. In the first section, an overviewof the techniques and implementation in accordance with one embodimentof the invention. In the next sections, an exemplary system andalgorithms for providing aspects of the present invention are discussed.Following this, an exemplary computer environment for the implementationand use of the invention is described.

In general, the present invention provides a computer-implemented methodin the form of computer-executable instructions programmed for allowinga consumer (hereinafter “user”) to order a print product including adigital image using a mobile computing device. The method may includeallowing for a camera on the mobile computing device to take a video ofa physical product being exhibited on a shelf or end cap, or in anotherlocation, in a retail store location, for example. It should beunderstood that the exhibited physical product (“exhibited product”) mayinclude, but is not limited to, a real-life three-dimensional productsuch as a calendar, mug, photo book, poster, blanket, phone case,pillow, clothing, or jewelry. The real-life three-dimensional productmay be a full-scale model of the product, or some other scaled versionthereof. The exhibited product may also be a two-dimensionalillustration of one or more products (e.g., calendar, mug, photo book,poster, blanket, phone case, pillow, clothing, or jewelry) displayed ona two-dimensional image print, such as a poster display board (e.g.,18″×24″). The use of the image print allows for the display of theproducts at multiple locations within a retail environment withouthaving to put any three-dimensional product products on display.

The method then operates to scan the exhibited product and use imageand/or object recognition to determine if the exhibited product is aproduct available to be used to fulfill a print order (“availableproduct”). In one aspect, the recognition may be performed by comparingat least one scanned feature (e.g., image) or attribute (e.g., size,shape, dimensions) of the exhibited product with one or more feature orattribute corresponding to available products stored in a memory. In theinstance that an image print is used to display a two-dimensionalillustration of the one or more exhibited products, it is the productsillustrated in the image print that are scanned for purposes of imageand/or object recognition. Upon determining that the exhibited productmatches an available product, an augmented reality system is used todisplay a digital overlay in the video and in association with theexhibited product being displayed on the mobile computing device. Thedigital overlay may include text, numbers, symbols, or any other type ofinformation, including but not limited to information related to theavailable product. It should also be understood that the digital overlaymay alternatively include no information. The method further allows forthe digital overlay to be selected by using a touch screen on the mobilecomputing device, for example. By selecting the digital overlay, themethod may allow for additional detailed information to be displayedrelated to the available product. Subsequent to the display of thedigital overlay, the method allows for a digital image to be selected bythe user. The digital image that is selected would be a digital imagethat the user desires to have disposed on the product to provide a printproduct. The method then may use a print product ordering system togenerate a print order thereafter allow for the fulfillment of the printproduct by a photofinishing system. Other aspects of the presentinvention will be described in more detail below.

Referring now to FIG. 1, an exemplary system that may be used toimplement the methods and aspects described herein is identified asreference number 100. System 100 may comprise one or more of a printproduct ordering server 102, a mobile computing device 104, and aphotofinishing system 106 that are in communication over a network 108.Network 108 may be any type of network, such as a wide area network(WAN) or local area network (LAN) through a wired or wirelessconnection. In one aspect as described below, server 102 and mobilecomputing device 104 operate in conjunction with one another toaccomplish the functionality provided by the methods described herein.However, it should be understood that it is also within the scope of thepresent invention to have mobile computing device 104 perform one ormore operations that are provided by server 102.

Server 102 includes a memory 110 that is configured for storing one ormore digital images 112 that are transferred or otherwise communicatedfrom mobile computing device 104 or some other remote computing device(e.g., desktop computing device) over network 108. The transfer ofdigital images to server 102 may also be from an external storagelocation, such as a remote social media server 114 or website (e.g.,INSTAGRAM® and IPHOTO®) or cloud-based memory location when such atransfer is authorized by a user or otherwise permitted using mobilecomputing device 104. Memory 110 may also include one or more digitalrepresentations of available products 116 that are available to be usedto produce a print product. The available products may include, but arenot limited to, an image print, card, invitation, frame, calendar,canvas, wall art, mug, photo book, poster, blanket, phone case, laptopsleeve, tablet sleeve, tote, pouch, coaster, placemat, plate, magnet,pillow, towel, clothing, jewelry, jewelry box, toy, and any otherproduct or substrate (wood, metal, paper, etc.) where an image can bedisposed thereon or associated therewith. A print product is defined asan available product including a digital image disposed thereon orassociated therewith, wherein the digital image may be provided by acustomer or obtained from another source. The utilization of the digitalrepresentations of available products 116 stored in memory 110 will bediscussed in more detail below.

Memory 110 may also include data representative (e.g., list) of one ormore features or attributes of each product 117 that is available to beused to produce a print product. A feature of an available product maybe, for example, a stock digital image that corresponds to an availableproduct. A attribute of an available product may be one or moredimensions or shapes that are associated with an available product. Thefeatures and attributes may be used in conjunction with image and/orobject recognition to identify an exhibited product using the methodsdescribed herein.

Server 102 may further include a processor that is configured forexecuting computer executable instructions that are provided to performthe operations and functions described in detail below.

Mobile computing device 104 may be any type of computing device that iscapable of being operated by a user 119, and communicating with server102 over network 108, such as, but not limited to, a smartphone, tablet,IPHONE, etc. Computing device 104 is “mobile” in that it can betransported or moved by user 119 into and around a retail store locationor any other location (hereinafter referred to as “retail store location120”). As such, mobile computing device 104 may be in communication withnetwork 108 via a WAN wireless connection (e.g., broadband) or a LANwireless connection (e.g., Wi-Fi), for example. Mobile computing device104 includes a camera 122 that is configured to capture a video of oneor more physical products 118 being exhibited anywhere in retail storelocation 120, including on a shelf or end-cap 124. It should beunderstood that the capturing of the video includes taking the video forreal-time viewing. In particular, mobile computing device 104 mayfurther include a user interface or display 126 (hereinafter “userinterface”) for displaying the real-time video of the one or moreexhibited products 118 to user 119 as camera 122 is being used to takethe video. In another aspect, the video that is captured by camera 122may be stored in a memory 128 of mobile computing device 104, forexample, for use with the system and method set forth herein. Memory 128may also be included in mobile computing device 104 to store one or moredigital images 130 that may be incorporated or otherwise used to producea print product. Digital images 130 may have been captured using camera122 of mobile computing device 104, downloaded from external storagelocation 114, or downloaded from memory 110, 112 of server 102.Likewise, digital images 130 may be uploaded to external storagelocation 114 or memory 110, 112 of server 102 so that they can be lateraccessed by system for generating a print product using the method ofthe present invention. It is also contemplated that the digitalrepresentations 116 of available products and/or data representative ofone or more features or attributes for each available product 117 bestored in memory 128 of mobile computing device 104 instead of, or inaddition to, storing them in memory 110 of server 102.

Mobile computing device 104 may also include a print product orderingsystem 132 stored in memory 128. Print product ordering system 132 maybe configured to associate at least one of the stored digital images112, 115, 130 (“digital image”) with a digital representation of anavailable print product 116 to form a digital representation of theresulting print product for display on user interface 126. Theassociation of the digital image and the digital representation of theavailable product provides user 119 with a visual representation of howthe digital image will look on an available product if it were orderedand fulfilled. Print product ordering system 132 may also operate toallow for an association of a digital image with an available productwithout displaying a visual representation of the resulting printproduct. The digital image 112, 115, 130 used by print product orderingsystem 132 may be obtained from memory 128 of mobile computing device104, memory 110 of server 102, or downloaded to memory 128 from externalstorage location 114.

Print product ordering system 132 is further configured to allow user119 to establish a print order including the selected print product andthe associated digital image to be disposed thereon. Once the printorder is established, print product ordering system 132 proceeds toallow for the payment of the print order and allow for the selection ofa delivery method. When it comes time to fulfill the print order byproducing the print product including the digital image, system 100 maycommunicate the print order to photofinishing system 106 over network108. The determination as to what type of printing device will be usedto fulfill the print order may be determined based on the capabilitiesof printing devices 106 a, 106 b and/or the supply of the printingsubstrates (selected print product) that are currently available toprinting devices 106 a, 106 b. It should be understood that printingdevices 106 a, 106 b illustrated in FIG. 1 are merely exemplary and maytake other forms depending on the print product selected by user 119.

Print product ordering system 132 may further include a paymentprocessing module that allows user 119 to provide payment for thefulfillment and/or delivery of the print order. For example, paymentprocessing module may be configured to use mobile computing device 214to communicate the necessary information (e.g., name, address, and/orcredit card information) to pay for the print order. Payment may beprovided through print product ordering server 102 or through a thirdparty payment processing system without passing any payment informationto the print product ordering server 102. Once a payment authorizationis received by print product ordering server 102, instructions may thenbe communicated to photofinishing system 106 so that the print order canbe fulfilled by an appropriate printing device 106 a, 106 b.

System 100 further includes an augmented reality system 134 that isprovided to facilitate the selection of product 118 during the use ofprint product ordering system 132. In one aspect, when camera 122 ofmobile computing device 104 is being used to capture a real-time videoof one or more physical products 118 being exhibited in retail storelocation 120, augmented reality system 134 is configured to scan thecaptured real-time video and utilize image and/or object recognition toidentify any available products that may be present in the capturedvideo.

In the instance that image recognition is utilized in augmented realitysystem 134, each of the products 118 being exhibited in retail storelocation 120 may include a stock image disposed thereon, such as, forexample, a picture or a text phrase (e.g., “personalize me”). The stockimages on each of the products 118 are preferably different in one ormore respects (e.g., different images, different font style for textphrases, etc.) to allow for better differentiation between the products118 and improve product identification success rate. Further, thedigital representations of available products 116 may each include oneor more features associated therewith, such as a stock digital image.For example, each of the digital representations of available products116 may be associated with a different stock digital image. As the videoof one or more physical products 118 being exhibited in retail storelocation 120 is captured, augmented reality system 134 may be configuredto scan each of the displayed physical products 118 to identify therespective stock images disposed thereon. Each of the respectiveidentified stock images are then compared with the stock digital imagesassociated with the digital representations of available products 116.If an identified stock image of a displayed physical product 118 matchesa stock digital image on a digital representation of an availableproduct 116, then augmented reality system 134 determines that therespective displayed physical product 118 corresponds to a respectivedigital representation of an available product 116. Augmented realitysystem 134 is then configured to retrieve a digital overlay 136associated with the respective digital representation of the availableproduct 116 corresponding to the identified physical product 118, anddisplay the retrieved digital overlay 136 in the video and inassociation with the physical product 118 being displayed on the userinterface 126. Digital overlay 136 may include any type of informationor no information. For example, digital overlay 136 may includeinformation such as, but not limited to, a description of the physicalproduct 118 in the form of text, video and/or audio description of theproduct, for example. The description of the product may include ageneral identification of the product, such as, but not limited to, “5×7frame,” “2018 calendar,” “mug,” etc. Other information related to theproduct, or not related to the product, may also be provided. Thedigital overlay 136 may be displayed in association with the physicalproduct 118 in the video on the user interface 126 by, for example,being superimposed on top of the displayed physical product 118, and/orabove, below, right, left, or diagonal of the displayed exhibitedphysical product 118. Further, augmented reality system 134 may beconfigured to display digital overlay 136 as a three-dimensionalrendered object in the video as if digital overlay 136 is actuallyphysically on top of the physical product 118 or otherwise associatedwith physical product 118 in the displayed video. As such, when user 119moves camera 122 when displaying physical product 118 on user interface126, the perspective appearance of digital overlay 136 on user interface126 changes along with the displayed physical product 118. This givesuser 119 the impression that the displayed digital overlay 136 isactually in the same environment as physical product 118 in retail storelocation 120. Data representative of each digital overlay 136 may bestored in memory 124 of mobile computing device 104 and/or memory 110 ofserver 102. Augmented reality system 134 is also configured for allowingdigital overlay 136 to be selected using user interface 126 of mobilecomputing device to display additional detailed information related orunrelated to the respective physical product 118, and/or allow user 119to proceed with ordering a print product made up of a productcorresponding to the selected physical product 118 and a digital imageto be selected by user 119 using the print order ordering system 132 asdescribed herein. The additional detailed information related to therespective physical product 118 may include characteristics of theproduct including, but not limited to, dimensions, shape, materials,cost, and other attributes related to the product.

It should be understood that augmented reality system 134 operates in asimilar manner as described above if object recognition is utilized toscan physical product 118 being displayed in the video, and compare thescanned physical product 118 with one or more attributes of the digitalrepresentations of available products 116. Instead of a respective stockdigital image being associated with each of the digital representationsof available products 116, the digital representations of availableproducts 116 would be associated with one or more attributes, such as,but not limited to, at least one dimension or shape that is unique to aparticular product that will allow augmented reality system 134 toidentify the physical product 118 being displayed in the video beingtaken in the retail store location 120. For example, each respectivedigital representation of available products 116 would be associatedwith certain dimensions. Therefore, if a physical product 118 displayedin a video taken by mobile computing device 104 is scanned and isdetermined to have dimensions of 3″×5″, for example, then thesedimensions would be compared with the dimensions associated with thedigital representations of available products 116 to identify whichavailable product(s) match these dimensions. It should be understoodthat the dimensions can be taken along one, two or three axes. Once itis determined which available product corresponds to the displayedphysical product 118, then the digital overlay associated with theidentified available product is displayed in the video and inassociation with the displayed physical product 118, as was previouslydescribed. This same methodology applies by using at least one shape ofthe physical product 118 displayed in the video during the objectrecognition process. Each digital representation of the availableproducts 116 could have a unique shape that is used to identify thephysical product 118 being displayed in the video. The shape could alsoinclude information related to the proportions of a given object. Forexample, one unique shape of an available product 116 could be arectangle with proportions of 3:5, which would be indicative of a 3×5frame product. Another shape could be two squares positioned adjacent toone another, which could be indicative of a calendar. It should beunderstood that the above mentioned examples are merely exemplary, andany other shape, proportion, dimension may be used to accomplish themethods described herein.

While print product ordering system 132 and augmented reality system 134is shown in FIG. 1 as being stored in memory 128 of mobile computingdevice 104, it should be understood that print product ordering system132 and/or augmented reality system 134 may be stored in memory 110 ofserver 102. Also, print product ordering system 132 and augmentedreality system 134 are not limited to any specific hardware or softwareconfiguration, but may rather be implemented as computer executableinstructions in any computing or processing environment, including indigital electronic circuitry or in computer hardware, firmware, devicedriver, or software.

FIG. 2 is a flow diagram showing an exemplary method 200 that may beimplemented using system 100 in accordance with one aspect of thepresent invention. FIGS. 3-12 are exemplary screen shots from mobilecomputing device 104 displayed on user interface 126 during theimplementation of method. The aforementioned screen shots will bereferred to throughout the discussion set forth below to illustrate anexemplary implementation of method 200.

Method 200 is computer-implemented and programmed for execution in acomputing environment for ordering a print product. As best seen in FIG.2, method 200 may comprise a step 202 of storing at least one digitalrepresentation of an available print product. As previously described,each digital representation of the available print products may beassociated with data representative of one or more feature or attribute(e.g., unique stock image, dimension and/or shape) that allows for theidentification of an exhibited physical product that is displayed in avideo using the mobile computing device 104. Thus, step 202 could alsoinclude storing of the data representative of the one or more featuresor attributes. It is also contemplated that the method provide for thestorage of the data representative of one or more feature or attributethat allows for identification of one or more available print products,without storing at least one digital representation of the one or moreavailable print products. Next, in step 204, method 200 allows forcamera 122 of mobile computing device 104 to be used to capture areal-time video 205 of at least one physical product 118 a, 118 b beingexhibited in retail store location 120, and displaying real-time video205 on user interface 126 of mobile computing device 104, as seen inFIG. 3.

Augmented reality system 134 is then utilized to scan video 205displayed on user interface 126 to identify any exhibited physicalproducts 118 a. 118 b displayed in video 205, and use image and/orobject recognition to compare the physical products 118 a, 118 b beingdisplayed in video 205 with features and/or attributes of availableproducts 116, at step 206. If it is determined that one or more ofphysical products 118 a, 118 b being displayed in video 205 correspondto certain digital representations of available products 116, augmentedreality system 134 is utilized to retrieve data representative of arespective digital overlay 136 that corresponds to the identifiedphysical products 118 a, 118 b stored in memory 128 and/or server 102.At step 208, method 200 includes using the retrieved data 136 to displaythe respective digital overlays 136 a, 136 b in association with therespective physical product 118 a, 118 b on user interface 126, as bestseen in FIG. 3. At step 210, method 200 further allows for the selectionof any of the digital overlays 136 a, 136 b being displayed in video205. For example, user 119 may select either digital overlay 136 a ordigital overlay 136 b by physically touching the respective digitaloverlay using a touch screen feature on user interface 126. Othermethods of selecting the respective digital overlay are alsocontemplated herein.

As seen in the progression from FIG. 5 to FIG. 6, and from FIG. 5 toFIG. 7, augmented reality system 134 may operate to associate (e.g.,superimpose, position above, below, right, left, or diagonal) digitaloverlays 136 a, 136 b with physical products 118 a, 118 b beingdisplayed in video 205, and allow for the perspective appearance ofdigital overlays 136 a, 136 b to change to match the perspectiveappearance of physical products 118 a, 118 b displayed on user interface126. The perspective appearance of both the physical products 118 a, 118b and the associated digital overlays 136 a, 136 b may change in unisonas the angle and/or position of camera 122 changes relative to physicalproducts 118 a, 118 b in retail store location 120. This feature givesuser 119 the impression that digital overlays 136 a, 136 b are actuallyin the same physical environment as physical products 118 a, 118 b toenhance the product ordering experience for user 119.

In one aspect, selecting digital overlay may result in the display ofadditional information related to the respective identified product. Forexample, selection of digital overlay 136 a may result in the display ofan additional window 211 a as best seen in FIG. 4, and selection ofdigital overlay 136 b may result in the display of an additional window211 b as best seen in FIG. 5. As previously mentioned, the additionaldetailed information related to the respective physical product 118 a,118 b may include characteristics of the product including, but notlimited to, dimensions, shape, materials, cost, and other attributesrelated to the product. In either of the additional windows 211 a, 211 ban order button 212 a, 212 b may be displayed to allow user 119 toproceed with ordering a print product. For example, a screen such asthat shown in FIG. 8 may be displayed on user interface 126 after orderbutton 212 b is selected, which allows user to proceed with selecting adigital image at step 214 by selecting the “Create Product” button 216.It should be understood that instead of displaying additional window 211and/or the screen shown in FIG. 8 after selecting a digital overlay, itis contemplated that method 200 proceed directly to allowing for theselection of a digital image at step 214.

At step 214, method 200 further allows for the selection of a digitalimage to be disposed on or otherwise incorporated with a selectedphysical product. For example, as best seen in FIG. 9, user 119 may usea button 216 that allows for the selection of one or more digitalimages. One or more digital images 218 may be retrieved from memory 110,114 and/or 128. As best seen in FIG. 10, for example, after digitalimage 218 is selected, a digital representation 220 corresponding to theproduct 118 b associated with the selected digital overlay 136 b may beoptionally displayed on user interface 126 including the selecteddigital image 218, at step 222, to show user 119 what the print productwould look like if fulfilled. While in the example shown in FIGS. 5-7shows physical product 118 b as including a unique stock image, itshould be understood that the digital representation 220 includes theselected digital image 218 in place of the unique stock image. Inanother aspect, augmented reality system 134 may be used to display theselected digital image 218 in augmented reality so that it issuperimposed on top of the stock image on, and/or above, below, right,left, or diagonal of, the exhibited physical product 118 a beingdisplayed in video 205. This allows the physical product 118 a beingdisplayed in video 205 to be shown in association with the selecteddigital image 218. Displaying digital representation 220 in the mannershown in FIG. 10, or in augmented reality within a displayed video, willallow user 119 to view the selected product (e.g., 11″×14″ frame) anddigital image 218 in a real environment to give user 119 an idea of whatthe print product will look like if fulfilled by photofinishing system106.

At step 224, method 200 may allow for digital image 218 to be modified.For example, the digital image may be modified through a function thatallows for cropping, panning, enlarging, and/or rotating. Digital image218 may also be modified to include changes to color characteristics,such as, but not limited to, brightness, saturation, hue, contrast, redeye, etc. Further, if the product selected by user 119 allows for theincorporation of multiple digital images (e.g., calendar), method 200may optionally allow for a layout 226 of the selected digital images tobe modified. Modification of digital image 218 may also include anyother type of editing feature, including but not limited to utilizingdigital image 218 in conjunction with one or more filters, templates, orother adjustments.

Once user 119 is finished selecting one or more digital images 218, an“Add to Cart” button 288 may be selected which results in an ordersummary 230 being displayed on user interface 126 including the displayof an order cost 232, as best seen in FIG. 11. A “Check Out” button 234may then be selected so that user 119 may provide payment information(e.g., name, credit card information) as well as shipping address 236and shipping method 238, as seen in FIG. 12. Method 200 further includesa step 240 of establishing a print order including an identification ofthe displayed exhibited physical product 118 b associated with theselected digital overlay 136 b (i.e., the selected product) and theselected digital image 218. The identification of the selected productcan be the name of the selected product, for example, 11″×14″ pictureframe, so photofinishing system 106 knows what product it needs to usein conjunction with the selected digital image to produce the printproduct. In the instance that the displayed exhibited physical product118 b associated with the selected digital overlay 136 b includes astock image disposed thereon, it should be understood that theidentification in the print order need not include any informationrelated to the stock image since the user selected digital image will beused to produce the print product. Further, it should be understood thatthe print order may also include an identification of the location ofdigital image 218 so that digital image 218 may be retrieved whenfulfilling the print order. In the alternative, the print order caninclude a data file containing digital image 218.

Method 200 further includes a step 242 of allowing for the fulfillmentof the print order to produce the print product. In particular, theprint order may be communicated from mobile computing device 104 and/orserver 102 to photofinishing system 106 over network 108. Theappropriate printing device 106 a, 106 b may then be used to produce theprint product depending on the capabilities of printing devices 106 a,106 b and the availability of such printing devices 106 a, 106 b. Forexample, the print product produced by the photofinishing system 106will include the selected digital image 218 disposed on the selectedphysical product that corresponds to the displayed exhibited physicalproduct 118 b associated with the selected digital overlay 136 b.

In another aspect, it should be understood that the methods set forthabove may be embodied in computer readable instructions stored on anon-transitory computer readable medium.

Having described the system and method of the present invention and anembodiment thereof, an exemplary computer environment for implementingthe described design and execution is presented next.

FIG. 13 shows an exemplary computing environment 300 that can be used toimplement any of the processing thus far described. Computingenvironment 300 may include one or more computers 312 (such as server102, mobile computing device 104, photofinishing system 106) comprisinga system bus 324 that couples a video interface 326, network interface328, a keyboard/mouse interface 334, and a system memory 336 (e.g.,memory 110, 114, 128) to a Central Processing Unit (CPU) 338. A display340 is connected to bus 324 by video interface 326 and provides the userwith a graphical user interface to view video 205, view and selectdigital overlays 136, and view, edit, and prepare a print order usingone or more digital images. The graphical user interface allows the userto enter commands and information into computer 312 using a keyboard 341and a user interface selection device 343, such as a mouse, touchscreen, or other pointing device. Keyboard 341 and user interfaceselection device 343 may be connected to bus 324 through keyboard/mouseinterface 334. The display 340 and user interface selection device 343are used in combination to form the graphical user interface whichallows the user to implement at least a portion of the presentinvention. Other peripheral devices may be connected to the remotecomputer through universal serial bus (USB) drives 345 to transferinformation to and from computer 312. For example, cameras andcamcorders may be connected to computer 312 through serial port 332 orUSB drives 345 so that data representative of a digital image, digitaloverlay, or video may be displayed on display 340 downloaded to systemmemory 336 or another memory storage device associated with computer 312such that the digital images may be subsequently printed on a selectedphysical product by photofinishing system 106.

The system memory 110, 128, 336 is also connected to bus 324 and mayinclude read only memory (ROM), random access memory (RAM), an operatingsystem 344, a basic input/output system (BIOS) 346, application programs348 and program data 350. The computer 312 may further include a harddisk drive 352 for reading from and writing to a hard disk, a magneticdisk drive 354 for reading from and writing to a removable magnetic disk(e.g., floppy disk), and an optical disk drive 356 for reading from andwriting to a removable optical disk (e.g., CD ROM or other opticalmedia). The computer 312 may also include USB drives 345 and other typesof drives for reading from and writing to flash memory devices (e.g.,compact flash, memory stick/PRO and DUO, SD card, multimedia card, smartmedia xD card), and a scanner 358 for scanning items such as digitalimages to be downloaded to computer 312. A hard disk drive interface 352a, magnetic disk drive interface 354 a, an optical drive interface 356a, a USB drive interface 345 a, and a scanner interface 358 a operate toconnect bus 324 to hard disk drive 352, magnetic disk drive 354, opticaldisk drive 356, USB drive 345 and scanner 358, respectively. Each ofthese drive components and their associated computer-readable media mayprovide computer 312 with non-volatile storage of computer-readableinstruction, program modules, data structures, application programs, anoperating system, and other data for computer 312. In addition, it willbe understood that computer 312 may also utilize other types ofcomputer-readable media in addition to those types set forth herein,such as digital video disks, random access memory, read only memory,other types of flash memory cards, magnetic cassettes, and the like.

Computer 312 may operate in a networked environment using logicalconnections with each of the system components described above. Networkinterface 328 provides a communication path 360 between bus 324 andnetwork 108, which allows, for example, a print order, digital image,unique payment link, unique identifier, input identifier, token ID,payment authorization, notification, inquiry, and the identified andselected substrate to be communicated through network 108 between therespective components of system 200. This type of logical networkconnection is commonly used in conjunction with a local area network(LAN). These items may also be communicated from bus 324 through acommunication path 362 to network 108 using serial port 332 and a modem364. Using a modem connection between the computer 312 and the othercomponents of system 100 is commonly used in conjunction with a widearea network (WAN). It will be appreciated that the network connectionsshown herein are merely exemplary, and it is within the scope of thepresent invention to use other types of network connections betweencomputer 312 and the other components of system 100 including both wiredand wireless connections.

In utilizing the system and method in accordance with the presentinvention, numerous advantages are realized. For example, leveraging theuse of a user's mobile computing device to select and order a printproduct eliminates the need for providing a photo kiosk in a retaillocation. This is beneficial in that it is a significant cost savingsnot to have to design, manufacture and provide photo kiosks to multipleretail locations. In many cases, all that would need to be provided toimplement the present invention are one or more representative samplesof exhibited products so a user can observe and evaluate a physicalrepresentation of the products that are available for purchase, and takea video of such exhibited products. The digital version of the availableproducts that are typically displayed to the user on a kiosk is not asbeneficial. Furthermore, it can be arranged so that all of the exhibitedproducts at the retail locations are the same, which will allow theimage and/or object recognition functionality to operate withconsistency across the retail locations.

From the foregoing, it will be seen that this invention is one welladapted to attain all the ends and objects hereinabove set forthtogether with other advantages which are obvious and which are inherentto the method and apparatus. It will be understood that certain featuresand sub combinations are of utility and may be employed withoutreference to other features and sub combinations. This is contemplatedby and is within the scope of the claims. Since many possibleembodiments of the invention may be made without departing from thescope thereof, it is also to be understood that all matters herein setforth or shown in the accompanying drawings are to be interpreted asillustrative and not limiting.

The constructions described above and illustrated in the drawings arepresented by way of example only and are not intended to limit theconcepts and principles of the present invention. As used herein, theterms “having” and/or “including” and other terms of inclusion are termsindicative of inclusion rather than requirement.

While the invention has been described with reference to preferredembodiments, it will be understood by those skilled in the art thatvarious changes may be made and equivalents may be substituted forelements thereof to adapt to particular situations without departingfrom the scope of the invention. Therefore, it is intended that theinvention not be limited to the particular embodiments disclosed as thebest mode contemplated for carrying out this invention, but that theinvention will include all embodiments falling within the scope andspirit of the appended claims.

What is claimed is:
 1. A computing system for providing a customizedprint product, the system comprising: a first memory for storingcomputer-executable instructions; a camera configured for capturing avideo; a display configured for displaying the video; and a processorconfigured for executing the computer-executable instructions to performthe following steps: a) utilizing the camera to display a video thatincludes an exhibited physical product; b) comparing one or more featureor attribute associated with the exhibited physical product beingdisplayed in the video with feature data or attribute data associatedwith an available product, wherein the feature data or attribute data isstored in a server memory of a server, wherein the server is incommunication with the processor over a network; c) upon determiningthat the displayed exhibited physical product corresponds to theavailable product, displaying a digital overlay in the video and inassociation with the displayed exhibited physical product; d) allowingfor the selection of the digital overlay being displayed in the video onthe display; and e) in response to the selection of the digital overlay,allowing for the selection of a digital image to customize the displayedexhibited physical product, wherein the selected digital image is notrepresentative of the displayed exhibited physical product, and whereinthe selected digital image is configured to be disposed on a selectedphysical product representative of the displayed exhibited physicalproduct associated with the selected digital overlay to provide thecustomized print product.
 2. A system in accordance with claim 1,wherein the feature data of the available product includes a stockdigital image, wherein the one or more feature of the exhibited physicalproduct is a stock image, wherein step c) includes scanning thedisplayed exhibited physical product to identify the stock imagedisposed thereon, and wherein the displayed exhibited physical productcorresponds to the digital representation of the available product whenthe stock image matches the stock digital image.
 3. A system inaccordance with claim 1, wherein the attribute data includes at leastone dimension of the available product, wherein step c) includesscanning the displayed exhibited physical product to identify the atleast one dimension of the displayed exhibited physical product, andwherein the displayed exhibited physical product corresponds to theavailable product when the at least one dimension of the displayedexhibited physical product matches the at least one dimension of theavailable product.
 4. A system in accordance with claim 1, wherein the \attribute data includes at least one shape of the available product,wherein step c) includes scanning the displayed exhibited physicalproduct to identify the at least one shape of the displayed exhibitedphysical product, and wherein the displayed exhibited physical productcorresponds to the available product when the at least one shape of thedisplayed exhibited physical product matches the at least one shape ofthe available product.
 5. A system in accordance with claim 1, whereinthe camera, the display, and the processor are included in a mobilecomputing device.
 6. A system in accordance with claim 5, wherein datarepresenting the digital overlay is stored in the server memory.
 7. Asystem in accordance with claim 5, wherein the digital image is storedin the first memory.
 8. A system in accordance with claim 7, wherein thefirst memory is included in the mobile computing device.
 9. A system inaccordance with claim 5, wherein the digital image is stored in a remotesocial media server, wherein the remote social media server is incommunication with the mobile computing device over the network.
 10. Asystem in accordance with claim 1, wherein the step of displaying thedigital overlay in association with the displayed exhibited physicalproduct includes superimposing the digital overlay on top of thedisplayed exhibited physical product.
 11. A system in accordance withclaim 10, wherein the digital overlay is displayed in the sameperspective as the exhibited physical product is displayed in the video.12. A system in accordance with claim 1, wherein the display includes atouch screen, and wherein the selection of the digital overlay isperformed using the touch screen.
 13. A system in accordance with claim1, wherein the video is a real-time video.
 14. A system in accordancewith claim 1, further comprising, subsequent to step e), the additionalstep of displaying a digital representation of the customized printproduct on the display, wherein the digital representation of thecustomized print product includes: the selected digital image; and adigital representation of the displayed exhibited physical productassociated with the selected digital overlay.
 15. A system in accordancewith claim 14, wherein the digital representation of the customizedprint product is displayed in the video superimposed on the exhibitedphysical product displayed on the video.
 16. A system in accordance withclaim 1, wherein the digital overlay includes a description of thedisplayed exhibited physical product.
 17. A system in accordance withclaim 1, wherein a plurality of digital images are stored in the firstmemory, and wherein the digital image is selected from the plurality ofdigital images.
 18. A system in accordance with claim 5, wherein aplurality of digital images are stored in a remote social media server,wherein the remote social media server is in communication with themobile computing device over the network, and wherein the digital imageis selected from the plurality of digital images.
 19. Acomputer-implemented method programmed for execution in a computingenvironment for providing a custom print product, the method comprising:a) displaying a video that includes an exhibited physical product; b)comparing one or more feature or attribute associated with the exhibitedphysical product being displayed in the video with feature data orattribute data associated with the available product, wherein thefeature data or the attribute data is stored in a memory; c) upondetermining that the displayed exhibited physical product corresponds tothe available product, displaying a digital overlay in the video and inassociation with the displayed exhibited physical product; d) allowingfor the selection of the digital overlay being displayed in the video;e) in response to the selection of the digital overlay, allowing for theselection of a digital image, wherein the selected digital image is notrepresentative of the displayed exhibited physical product, and whereinthe selected digital image is configured to be disposed on a selectedphysical product representative of the displayed exhibited physicalproduct associated with the selected digital overlay to provide thecustomized print product.
 20. A computing system for providing acustomized print product, the system comprising: a first memory forstoring computer-executable instructions; a camera configured forcapturing a video; a display configured for displaying the video; and aprocessor configured for executing the computer-executable instructionsto perform the following steps: a) utilizing the camera to display avideo that includes an exhibited physical product; b) comparing one ormore feature or attribute associated with the exhibited physical productbeing displayed in the video with feature data or attribute dataassociated with an available product, wherein the feature data orattribute data is stored in the first memory of a server; c) upondetermining that the displayed exhibited physical product corresponds tothe available product, displaying a digital overlay in the video and inassociation with the displayed exhibited physical product; d) allowingfor the selection of the digital overlay being displayed in the video onthe display; and e) in response to the selection of the digital overlay,allowing for the selection of a digital image to customize the displayedexhibited physical product, wherein the selected digital image is notrepresentative of the displayed exhibited physical product, and whereinthe selected digital image is configured to be disposed on a selectedphysical product representative of the displayed exhibited physicalproduct associated with the selected digital overlay to provide thecustomized print product.
 21. A computing system for providing acustomized print product, the system comprising: a first memory forstoring computer-executable instructions; a camera configured forcapturing a video; a display configured for displaying the video; and aprocessor configured for executing the computer-executable instructionsto perform the following steps: a) utilizing the camera to display avideo that includes an exhibited physical product; b) comparing one ormore feature or attribute associated with the exhibited physical productbeing displayed in the video with one or more feature or attributeassociated with a digital representation of an available product storedin the first memory; c) upon determining that the displayed exhibitedphysical product corresponds to the digital representation of theavailable product, displaying a digital overlay in the video and inassociation with the displayed exhibited physical product; d) allowingfor the selection of the digital overlay being displayed in the video onthe display; and e) in response to the selection of the digital overlay,allowing for the selection of a digital image to customize the displayedexhibited physical product, wherein the selected digital image is notrepresentative of the displayed exhibited physical product, and whereinthe selected digital image is configured to be disposed on a selectedphysical product representative of the displayed exhibited physicalproduct associated with the selected digital overlay to provide thecustomized print product.